Image processing method, image processing apparatus and electronic device

ABSTRACT

It is provided according to the present application an image processing method applied to an electronic device equipped with a binocular camera. The image processing method includes: calculating a motion direction of a first image and a motion direction of a second image, wherein the first image and second image are captured by two cameras of the binocular camera respectively; performing deblurring on the first image and the second image; and combining the images which are already deblurred to obtain a definite three-dimensional perspective image. According to the image processing method, a blurry image generated during imaging is deblurred, the blurring caused by movement or the like may be weakened, the definition of the deblurred image is better than the definition of the blurred image, thereby leading to a better imaging result.

CROSS-REFERENCES TO RELATED APPLICATIONS

The present application claims priority to Chinese Patent ApplicationNo. 201410497963.7, filed with the State Intellectual Property Office ofPeople's Republic of China on Sep. 25, 2014 entitled “IMAGE PROCESSINGMETHOD, IMAGE PROCESSING APPARATUS AND ELECTRONIC DEVICE”, the contentof which is incorporated herein by reference in its entirety.

FIELD

The disclosure generally relates to the field of an electronic device,and in particular to an image processing method, an image processingapparatus and an electronic device.

BACKGROUND

With the development of imaging technology, more and more electronicdevices are provided with an imaging function. Some electronic devicesare equipped with binocular cameras to ensure a real imaging result. Thebinocular camera is a camera group consisting of two cameras simulatinghuman eye imaging, and can combine the images captured respectively bythe two cameras to obtain a three-dimensional perspective image.

However, the captured image may be blurry due to that a hand of a usertrembles or various external movements occur for example a captureobject moves when the user captures the image, thereby leading to a badimaging result.

SUMMARY

In view of this, it is provided an image processing method according tothe disclosure, for solving the problem that the captured image may beblurry due to that a movement occurs when capturing the image.

To achieve the above object, there are provided following technicalsolutions according to the disclosure.

According to an aspect of the present disclosure, an image processingmethod is provided, the method includes:

-   -   capturing a first image and a second image using an imaging        device;    -   calculating a motion direction of the first image and a motion        direction of the second image;    -   performing deblurring on the first image based on the motion        direction of the first image to obtain a third image, and        performing deblurring on the second image based on the motion        direction of the second image to obtain a fourth image; and    -   combining the third image with the fourth image to obtain a        fifth image.

Optionally, in the above image processing method, the calculating themotion direction of the first image and the motion direction of thesecond image is based on a preset motion algorithm includes:

-   -   analyzing the first image based on a preset depth analyzing rule        to obtain a first depth image, and analyzing the second image        based on the preset depth analyzing rule to obtain a second        depth image; and    -   estimating the first depth image based on a preset estimating        algorithm to obtain a first motion direction of a pixel in the        first image, and estimating the second depth image based on the        preset estimating algorithm to obtain a second motion direction        of a pixel in the second image.

Optionally, in the above image processing method, the analyzing thefirst image based on the preset depth analyzing rule to obtain the firstdepth image and analyzing the second image based on the preset depthanalyzing rule to obtain the second depth image includes:

-   -   selecting one of the first image and the second image as a first        base image with the other one as a first reference image;    -   selecting a pixel in the first base image;    -   searching for a corresponding pixel in the first reference image        that matches with the selected pixel in the first base image;    -   determining a depth value of the selected pixel by using a        preset depth algorithm based on a position of the selected pixel        in the first base image, a position of the corresponding pixel        in the first reference image and a spatial parameter of the        imaging device;    -   calculating the depth value of each of the pixels in the first        base image other than the selected pixels; and    -   deriving a depth image of the first base image based on the        depth value of each of the pixels in the first base image.

Optionally, in the above image processing method, the estimating thefirst depth image based on the preset estimating algorithm to obtain thefirst motion direction of a pixel in the first image and estimating thesecond depth image based on the preset estimating algorithm to obtainthe second motion direction of a pixel in the second image includes:

-   -   selecting one of the first image and the second image as a        second base image and setting the depth image of the second base        image as a base depth image;    -   determining pixels with the same depth value in the second base        image and positions of the pixels with the same depth value in        the second base image based on the base depth image;    -   calculating the motion direction of the pixels with the same        depth value in the second base image based on a preset motion        direction estimating algorithm in conjunction with information        about the positions of the pixels with the same depth value in        the second base image; and    -   calculating the motion direction of each of the pixels in the        second base image other than the pixels with the same depth        value.

Optionally, in the above image processing method, the performingdeblurring on the first image based on the preset deblurring rule inconjunction with the first motion direction to obtain the third imageand performing deblurring on the second image based on the presetdeblurring rule in conjunction with the second motion direction toobtain the fourth image includes:

-   -   performing deconvolution calculation on the pixel in the first        image to obtain the third image based on a preset blurring        kernel model in conjunction with the first motion direction; and    -   performing the deconvolution calculation on the pixel in the        second image to obtain the fourth image based on the preset        blurring kernel model in conjunction with the second motion        direction.

According to an aspect of the present disclosure, an image processingapparatus is provided, which includes:

-   -   an imaging device for capturing a first image and a second        image;    -   a calculating module configured to calculate a motion direction        of the first image and a motion direction of the second image;    -   a deblurring module configured to perform deblurring on the        first image based on the motion direction of the first image to        obtain a third image, and perform deblurring on the second image        based on the motion direction of the second image to obtain a        fourth image; and    -   a combining module configured to combine the third image with        the fourth image to obtain a fifth image.

Optionally, in the above image processing apparatus, the calculatingmodule includes:

-   -   a first analyzing unit, configured to analyze the first image        based on a preset depth analyzing rule to obtain a first depth        image, and to analyze the second image based on the preset depth        analyzing rule to obtain a second depth image; and    -   a first calculating unit, configured to estimate the first depth        image based on a preset estimating algorithm to obtain a first        motion direction of a pixel in the first image, and to estimate        the second depth image based on the preset estimating algorithm        to obtain a second motion direction of a pixel in the second        image.

Optionally, in the above image processing apparatus, the first analyzingunit includes:

-   -   a first selecting subunit, configured to select one of the first        image and the second image as a first base image with the other        one as a first reference image, and to select a pixel from the        pixels in the first base image;    -   a searching subunit, configured to search for a corresponding        pixel in the first reference image matching with the selected        pixel;    -   a first calculating subunit, configured to determine a depth        value of the selected pixel by using a preset depth algorithm        based on a position of the selected pixel in the first base        image, a position of the corresponding pixel in the first        reference image and a spatial parameter of the binocular camera,        and to calculate the depth value of each of the pixels in the        first base image other than the selected pixel; and    -   a drawing subunit, configured to draw the depth image of the        first base image based on the depth value of each of the pixels        in the first base image.

Optionally, in the above image processing apparatus, the firstcalculating unit includes:

-   -   a second selecting subunit, configured to select one of the        first image and the second image as a second base image, and to        set the depth image of the second base image as a base depth        image;    -   a determining subunit, configured to determine pixels with the        same depth value in the second base image and positions of the        pixels with the same depth value in the second base image based        on the base depth image; and    -   a second calculating subunit, configured to calculate the motion        direction of the pixels with the same depth value in the second        base image based on a preset motion direction estimating        algorithm in conjunction with information about the positions of        the pixels with the same depth value in the second base image,        and to calculate the motion direction of each of the pixels in        the second base image other than the pixels with the same depth        value.

Optionally, in the above image processing apparatus, the deblurringmodule is further configured to:

-   -   perform deconvolution calculation on the pixel in the first        image to obtain the third image based on a preset blurring        kernel model in conjunction with the first motion direction; and    -   perform the deconvolution calculation on the pixel in the second        image to obtain the fourth image based on the preset blurring        kernel model in conjunction with the second motion direction.

According to an aspect of the present disclosure, an electronic deviceis provided, which includes a binocular camera and any one of the aboveimage processing apparatuses.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings to be used in the description of theembodiments or the prior art are described briefly as follows, so thatthe technical solutions according to the embodiments of the presentdisclosure or according to the prior art become clearer. It is apparentthat the accompanying drawings in the following description are a few ofembodiments of the present disclosure. For those skilled in the art,other accompanying drawings may be obtained according to theseaccompanying drawings without any creative work.

FIG. 1 is a flow chart of a first embodiment of an image processingmethod according to the application;

FIGS. 2A and 2B show a first image and a second image in the firstembodiment of the image processing method according to the application;

FIG. 3 shows a fifth image in the first embodiment of the imageprocessing method according to the application;

FIG. 4 is a flow chart of a second embodiment of the image processingmethod according to the application;

FIG. 5 is a flow chart of a third embodiment of the image processingmethod according to the application;

FIG. 6 is a schematic diagram of calculating an actual physical positioncorresponding to a pixel in the third embodiment of the image processingmethod according to the application;

FIG. 7 shows a first depth image in the third embodiment of the imageprocessing method according to the application;

FIG. 8 is a flow chart of a fourth embodiment of the image processingmethod according to the application;

FIG. 9 is a schematic diagram of a motion direction of the depth imagein the fourth embodiment of the image processing method according to theapplication;

FIG. 10 is a schematic structural diagram of a first embodiment of animage processing apparatus according to the application;

FIG. 11 is a schematic structural diagram of a second embodiment of theimage processing apparatus according to the application;

FIG. 12 is a schematic structural diagram of a first analyzing unit in athird embodiment of the image processing apparatus according to theapplication; and

FIG. 13 is a schematic structural diagram of a first calculating unit ina fourth embodiment of the image processing apparatus according to theapplication.

DETAILED DESCRIPTION

The technical solution according to the embodiments of the presentdisclosure are described clearly and completely as follows inconjunction with the accompany drawings in the embodiments of thepresent disclosure. It is obvious that the described embodiments areonly a few of the embodiments according to the present disclosure. Allthe other embodiments obtained by those skilled in the art based on theembodiments in the present disclosure without any creative work belongto the scope of protection of the present disclosure.

As is shown in FIG. 1, which is a flow chart of a first embodiment of animage processing method according to the application. The imageprocessing method is applied to an electronic device equipped with abinocular camera. The electronic device may be a desktop, a laptop, atablet computer, a mobile phone, a smart TV, a smart watch, a wearabledevice or the like.

The image processing method may include step S101 to step S104.

In step S101, a first image and a second image captured by an imagingdevice (e.g. the binocular camera) are received.

One of two cameras of the binocular camera equipped in the electronicdevice captures a capture object for one frame as the first image, andthe other one of the two cameras of the binocular camera capturessimultaneously the same capture object for one frame as the secondimage.

The first image is corresponding to a position of the one of the twocameras of the binocular camera, and the second image is correspondingto a position of the other one of the two cameras of the binocularcamera. The first image and the second image are similar but different.

In practice, the images captured may be blurry due to that theelectronic device trembles or the capture object moves when capturingthe images.

As is shown in FIG. 2, which shows the first image and the second imagein the first embodiment of the image processing method. Figure (a) isthe first image captured by the one of the two cameras of the binocularcamera, and Figure (b) is the second image captured by the other one ofthe two cameras of the binocular camera. The first image and the secondimage are blurry, content of the two images are similar but different.

In step S102, a motion direction of the first image and a motiondirection of the second image are calculated based on a preset motionalgorithm.

First to be noted that an actual image blurring principle is thatdifferent depth values of the pixels correspond to different motiondirections of the pixels which cause the blurring. Therefore, the motiondirection of the blurry image is calculated, and deblurring is performedon the blurry image based on the motion direction.

The motion algorithm is preset, and the motion direction of the imagemay be calculated based on the preset motion algorithm.

Specifically, the motion direction of the first image is calculatedbased on information of the first image, and the motion direction of thesecond image is calculated based on information of the second image.

The process of calculating the motion direction is described in detailin later embodiments, which is not described herein.

In step S103, deblurring is performed on the first image based on apreset deblurring rule in conjunction with the motion direction of thefirst image to obtain a third image, and deblurring is performed on thesecond image based on the preset deblurring rule in conjunction with themotion direction of the second image to obtain a fourth image.

The deblurring rule is preset, and the preset deblurring rule matcheswith the motion direction. The image the motion direction of which isalready determined may be deblurred based on the preset deblurring ruleto obtain a clearer image.

Specifically, deblurring is performed on the first image based on thepreset deblurring rule in conjunction with the motion direction of thefirst image to obtain the third image, and deblurring is performed onthe second image based on the preset deblurring rule in conjunction withthe motion direction of the second image to obtain the fourth image.

The process of performing deblurring on the image is described in detailin later embodiments, which is not described herein.

In step S104, the third image is combined with the fourth image based ona preset combining rule to obtain a fifth image.

The definition of the third image is better than the definition of thefirst image, and the definition of the fourth image is better than thedefinition of the second image.

Combining the third image and the fourth image to obtain the fifth imageincludes: acquiring a first pixel in the third image and informationabout a position of the first pixel, and determining the capture objectcorresponding to the first pixel; searching for a second pixel in thefourth image matching with the capture object corresponding to the firstpixel based on the capture object corresponding to the first pixel, anddetermining information about a position of the second pixel, where thefirst pixel and the second pixel constitute a corresponding pixel group;calculating a position deviation of corresponding pixel groupconstituted of the first pixel and the second pixel based on a presetalgorithm in conjunction with the information about the position of thefirst pixel and the information about the position of the second pixel;calculating the position deviation of each corresponding pixel groupbetween the third image and the fourth image; restoring depthinformation in a three-dimensional coordinate based on the positiondeviation of each corresponding pixel group; and combining thecorresponding pixels in the third image and the fourth image based onthe depth information of each corresponding pixel group to obtain thefifth image.

As is shown in FIG. 3, which shows the fifth image obtained by combiningthe first image and the second image in FIGS. 2A and 2B. The fifth imageis more definite than the first image and the second image, and may leadto a strong three-dimensional feeling.

In summary, it is provided an image processing method according to theembodiment. According to the image processing method, a motion directionof a first image and a motion direction of a second image are calculatedbased on a preset motion algorithm, where the first image is captured byone of two cameras of the binocular camera and the second image iscaptured by the other one of the two cameras of the binocular camera;deblurring is performed on the first image and the second image based ona preset deblurring rule; and the images which are already deblurred arecombined to obtain a clear three-dimensional perspective image.According to the image processing method, a blurry image generatedduring imaging is deblurred, the blurring caused by movement or the likemay be weakened, the sharpness of the deblurred image is better than thesharpness of the blurred image, thereby leading to a better imagingresult.

As is shown in FIG. 4, which is a flow chart of a second embodiment ofthe image processing method according to the application. The imageprocessing method may include step S401 to step S405.

In step S401, a first image and a second image captured by the binocularcamera are received.

Step S401 is the same as step S101 in the first embodiment of the imageprocessing method, which is not described herein.

In step S402, the first image is analyzed based on a preset depthanalyzing rule to obtain a first depth image, and the second image isanalyzed based on the preset depth analyzing rule to obtain a seconddepth image.

The depth refers to a spatial distance between the capture object in theimage and the camera by which the image is captured.

The depth analyzing rule is preset in the electronic device, the firstimage is analyzed based on the preset depth analyzing rule to obtain thefirst depth image of the first image, and the second image is analyzedbased on the preset depth analyzing rule to obtain the second depthimage of the second image.

Specifically, analyzing the first image based on the preset depthanalyzing rule to obtain the first depth image includes: selecting apixel A from the pixels in the first image; searching for a pixel A′ inthe second image matching with the pixel A; determining an actualphysical position corresponding to the pixel A by using thetriangulation algorithm based on a position of the pixel A in the firstimage, a position of the pixel A′ in the second image and a spatialparameter of the binocular camera; calculating a depth value of thepixel A in the first image based on the distance between the actualphysical position corresponding to the pixel A and the position of thepixel A in the first image; calculating the depth value of each of thepixels in the first image other than the pixel A; and drawing the firstdepth image based on the depth value of each of the pixels in the firstimage.

Specifically, analyzing the second image based on the preset depthanalyzing rule to obtain the second depth image includes: selecting apixel B from the pixels in the second image; searching for a pixel B′ inthe first image matching with the pixel B; determining an actualphysical position corresponding to the pixel B by using thetriangulation algorithm based on a position of the pixel B′ in the firstimage, a position of the pixel B in the second image and the spatialparameter of the binocular camera; calculating a depth value of thepixel B in the second image based on the distance between the actualphysical position corresponding to the pixel B and the position of thepixel B in the second image; calculating the depth value of each of thepixels in the second image other than the pixel B; and drawing thesecond depth image based on the depth value of each of the pixels in thesecond image.

The spatial parameters of the binocular camera include the distancebetween the centers of two cameras or the like.

It should be noted that the method for calculating the depth image ofeach image in the embodiment is not limited to the described method. Inpractice, the method for calculating the depth image may be othermethods capable of obtaining the depth image of the image, which is notlimited herein.

In step S403, the first depth image is estimated based on a presetestimating algorithm to obtain a first motion direction of a pixel inthe first image, and the second depth image is estimated based on thepreset estimating algorithm to obtain a second motion direction of apixel in the second image.

Each pixel in the image corresponds to a depth value, the depth value isa comparable value, and the depth values of the pixels are compared todetermine whether the pixels are the same pixel.

The estimating algorithm is preset, and generally the preset estimatingalgorithm is the blind deconvolution algorithm.

Specifically, the motion direction may be calculated based on the pixelswith the same depth value, the process of calculating the motiondirection is described in detail in later embodiments, which is notdescribed herein.

In step S404, deblurring is performed on the first image based on apreset deblurring rule in conjunction with the motion direction of thefirst image to obtain a third image, and deblurring is performed on thesecond image based on the preset deblurring rule in conjunction with themotion direction of the second image to obtain a fourth image.

In step S405, the third image is combined with the fourth image based ona preset combining rule to obtain a fifth image.

Step S404 and step S405 are respectively the same as step S103 and stepS104 in the first embodiment of the image processing method, which arenot described herein.

In summary, according to the image processing method provided in theembodiment, the depth image of the first image and the depth image ofthe second image are determined based on the preset depth analyzingrule, the first motion direction of the pixel in the first image and thesecond motion direction of the pixel in the second image are obtainedbased on the preset estimating algorithm in conjunction with the depthimages, and deblurring is performed on the blurry image based on themotion direction of blurry image. According to the image processingmethod, the motion direction of the pixels in the image is calculatedbased on the depth image of the image, deblurring is performed on ablurry image generated during imaging, the blurring caused by movementor the like may be weakened, the definition of the obtained image isbetter than the definition of the blurry image, thereby leading to abetter imaging result.

As is shown in FIG. 5, which is a flow chart of a third embodiment ofthe image processing method according to the application. The imageprocessing method may include step S501 to step S509.

In step S501, a first image and a second image captured by the binocularcamera are received.

Step S501 is the same as step S401 in the second embodiment of the imageprocessing method, which is not described herein.

In step S502, one of the first image and the second image is selected asa first base image with the other one as a first reference image, and apixel is selected from the pixels in the first base image.

Any one of the first image and the second image is selected as the firstbase image, and the other one is as the first reference image. A motiondirection of the pixel in the first base image is calculated based onthe first base image and the first reference image.

It should be noted that a process from step S502 to step S505 is theprocess of calculating the motion direction of any one of the pixels inthe first base image.

Each image includes a plurality of pixels, and any one of the pixels inthe first base image is selected as a base pixel for determining themotion direction.

In step S503, a corresponding pixel in the first reference imagematching with the selected pixel is searched.

The first base image and the first reference image are captured by twocameras of the binocular camera, the two images are similar but there isa little angle difference between the two images. In each of the twoimages, there is image content corresponding to the same capture object.

Therefore, a pixel in the first reference image corresponding to theselected pixel may be found, a position of the corresponding pixel inthe first reference image is different from a position of the selectedpixel in the first base image.

Step S503 includes: determining a capture object corresponding to theselected pixel in the first base image; the corresponding pixel in thefirst reference image matching with the selected pixel is searched basedon the capture object, and the matching pixel is set as thecorresponding pixel matching with the selected pixel.

Specifically, in the case that the selected pixel and the correspondingpixel matching with the selected pixel are for the same capture object(or capture content), a matching degree between the selected pixel andthe corresponding pixel is high.

In practice, the process of determining the matching degree between theselected pixel and the corresponding pixel may include: firstlydetermining the position of the selected pixel in the first base image,and searching for a region in the first reference image close to theposition of the selected pixel in the first base image based on theposition of the selected pixel in the first base image; and thencomparing each of the pixels in the region with the selected pixel inthe first base image based on various information about the capturecontent such as color of the pixel or the like, to obtain the matchingdegree, the pixel with the highest matching degree is the correspondingpixel matching with the selected pixel.

In step S504, a depth value of the selected pixel is determined by usinga preset depth algorithm based on the position of the selected pixel inthe first base image, the position of the corresponding pixel in thefirst reference image and a spatial parameter of the binocular camera.

The position of the selected pixel in the first base image such as acoordinate position of the selected pixel in the first base image isrecorded when the selected pixel is selected.

The position, such as a coordinate position, of the corresponding pixelin the first reference image determined in step S503 is determined.

The depth algorithm is preset, and an actual physical positioncorresponding to the selected pixel is calculated by using the presetdepth algorithm based on the position of the selected pixel in the firstbase image, the position of the corresponding pixel in the firstreference image and the spatial parameter of the binocular camera. Theactual physical position corresponding to the selected pixel refers tothe position of the capture object corresponding to the selected pixelin three-dimensional space.

As is shown in FIG. 6, which is a schematic diagram of calculating theactual physical position corresponding to the pixel. C1 and C2 arepositions of two cameras of the binocular camera, in an image 601, apixel x1 is a certain pixel in the image captured by the camera at C1,and in an image 602, a pixel x2 is a certain pixel in the image capturedby the camera at C2, the pixel x1 and the pixel x2 are pixels for acapture object X, the capture object X locates at an intersection pointbetween a straight line defined by C1 and the pixel x1 and a straightline defined by C2 and the pixel x2, the distance between C1 and C2 isknown, the distance between C1 and the pixel x1 is determined when thepixel x1 is selected, and the distance between C2 and the pixel x2 isdetermined when the pixel x2 is selected. The distance between thecapture object X and the pixel x1 and the distance between the captureobject X and the pixel x2 may be calculated according to thetrigonometric function. The actual physical position corresponding tothe pixel x1 is calculated based on the coordinates of C1, C2, the pixelx1 and the pixel x2.

Specifically, the actual physical position refers to the distancebetween the capture object X and the electronic device, such as thedistance between the capture object X and the midpoint of C1 and C2.

It should be noted that in the case that an xy coordinate system isdefined with the straight line defined by C1 and C2 being an x-axis andthe direction from C1 to C2 being a positive direction of the x-axis andthe direction from a point in the straight line defined by C1 and C2 tothe capture object X being a positive direction of a y-axis, thecoordinate of the capture object X may be determined as the actualphysical position corresponding to the pixel x1.

It should be noted that the actual physical position corresponding tothe pixel x1 is the same as the actual physical position correspondingto the pixel x2 in the image 602. In the case that the image 602 is setas the first base image, the actual physical position corresponding tothe pixel x2 is not required to be calculated, because the actualphysical position corresponding to the pixel x2 is the same as theactual physical position corresponding to the pixel x1 calculated basedon the image 601.

The depth value of the pixel x1 refers to the distance between thecapture object X corresponding to the pixel x1 and the camera forcapturing the image to which the pixel x1 belongs.

The actual physical position corresponding to the pixel x1 isdetermined, the distance between the actual physical positioncorresponding to the pixel x1 and the camera for capturing the image towhich the pixel x1 belongs is calculated based on the actual physicalposition corresponding to the pixel x1 and the position of the camerafor capturing the image to which the pixel x1 belongs, thereby the depthvalue of the pixel x1 is calculated.

As is shown in FIG. 6, in the case that the distance between the captureobject X to the straight line defined by C1 and C2 is known, thedistance between the capture object X to C1 may be calculated accordingto the triangulation algorithm, thereby the depth value of the pixel x1in the first base image is calculated.

In step S505, the depth value of each of the pixels in the first baseimage other than the selected pixel is calculated.

The above process from step S502 to step S505 is the process ofcalculating the depth value of any one of the pixels in the first baseimage. The depth value of each of the pixels in the first base imageother than the selected pixel is calculated based on the process fromstep S502 to step S505.

It should be noted that, the first image is selected as the first baseimage and the depth value of each of the pixels in the first base imageis calculated. In the case that the second image is selected as thefirst base image, if there is a pixel in the second image matching witha pixel in the first image, the actual physical position correspondingto the pixel in the second image is not required to be calculated,because the actual physical position corresponding to the pixel in thesecond image is the same as the actual physical position correspondingto the matching pixel in the first image.

In step S506, the depth image of the first base image is drawn based onthe depth value of each of the pixels in the first base image.

The depth value of each of the pixels in the first base image isdetermined in above steps. The depth image of the first base image isdrawn based on the depth value of each of the pixels in the first baseimage.

Specifically, a gray value of the pixel is used to indicate the depthvalue of the pixel, the greater the depth value of the pixel is, thegreater the gray value of the pixel is.

FIG. 7 shows a first depth image of the first image in FIG. 2A.

The first image is taken as an example, the process from step S502 tostep S506 includes: selecting a pixel A from the pixels in the firstimage; searching for a pixel A′ in the second image matching with thepixel A; determining an actual physical position corresponding to thepixel A by using the triangulation algorithm based on a position of thepixel A in the first image, a position of the pixel A′ in the secondimage and a spatial parameter of the binocular camera; calculating adepth value of the pixel A in the first image based on the distancebetween the actual physical position corresponding to the pixel A andthe position of the pixel A in the first image; calculating the depthvalue of each of the pixels in the first image other than the pixel A;and drawing the first depth image based on the depth value of each ofthe pixels in the first image.

The second image is taken as an example, the process from step S502 tostep S507 includes: selecting a pixel B from the pixels in the secondimage; searching for a pixel B′ in the first image matching with thepixel B; determining an actual physical position corresponding to thepixel B by using the triangulation algorithm based on a position of thepixel B′ in the first image, a position of the pixel B in the secondimage and the spatial parameter of the binocular camera; calculating adepth value of the pixel B in the second image based on the distancebetween the actual physical position corresponding to the pixel B andthe position of the pixel B in the second image; calculating the depthvalue of each of the pixels in the second image other than the pixel B;and drawing the second depth image based on the depth value of each ofthe pixels in the second image.

In step S507, the first depth image is estimated based on a presetestimating algorithm to obtain a first motion direction of the pixel inthe first image, and the second depth image is estimated based on thepreset estimating algorithm to obtain a second motion direction of thepixel in the second image.

In step S508, deblurring is performed on the first image based on apreset deblurring rule in conjunction with the motion direction of thefirst image to obtain a third image, and deblurring is performed on thesecond image based on the preset deblurring rule in conjunction with themotion direction of the second image to obtain a fourth image.

In step S509, the third image is combined with the fourth image based ona preset combining rule to obtain a fifth image.

Step S507 to step S509 are respectively the same as step S403 to stepS405 in the second embodiment of the image processing method, which arenot described herein.

In summary, according to the image processing method provided in theembodiment, the depth value of each of the pixels in the image iscalculated, the depth image of the image is drawn, the motion directionof the pixel in the image is determined based on the depth image,deblurring is performed on the image, the images which are alreadydeblurred are combined to obtain a clear three-dimensional perspectiveimage. According to the image processing method, deblurring is performedon a blurry image generated during imaging, the blurring caused bymovement or the like may be weakened, the definition of the deblurredimage is better than the definition of the blurry image, thereby leadingto a better imaging result.

As is shown in FIG. 8, which is a flow chart of a fourth embodiment ofthe image processing method according to the application. The imageprocessing method may include step S801 to step S808.

In step S801, a first image and a second image captured by the binocularcamera are received.

In step S802, the first image is analyzed based on a preset depthanalyzing rule to obtain a first depth image, and the second image isanalyzed based on the preset depth analyzing rule to obtain a seconddepth image.

Step S801 and step S802 are respectively the same as step S401 and stepS402 in the second embodiment of the image processing method, which arenot described herein.

In step S803, one of the first image and the second image is selected asa second base image and the depth image of the second base image is setas a base depth image.

Any one of the first image and the second image is selected as thesecond base image and the depth image of the second base image is set asthe base depth image. The motion direction of the pixel in the secondbase image is calculated based on the second base image and the basedepth image of the second base image.

It should be noted that a process from step S804 to step S805 is theprocess of calculating the motion direction of any one of the pixels inthe second base image.

In step S804, pixels with the same depth value in the second base imageand positions of the pixels with the same depth value in the second baseimage are determined based on the base depth image.

An actual image blurring principle is that different depth values of thepixels correspond to different motion directions of the pixels whichcause the blurring. In the present embodiment, the motion direction ofthe pixel is calculated based on the depth value of the pixel, andspecifically, the motion direction of the pixel is calculated based onthe pixels with the same depth value.

Specifically, the pixels with the same depth value in the second baseimage and positions of the pixels with the same depth value in thesecond base image are determined based on the base depth image. Thepixels with the same depth value in the second base image constitute apixel group with the same depth value, and the pixel group includes aplurality of pixels.

In step S805, the motion direction of the pixels with the same depthvalue in the second base image is calculated based on a preset motiondirection estimating algorithm in conjunction with information about thepositions of the pixels with the same depth value in the second baseimage.

The preset motion direction estimating algorithm is the blinddeconvolution algorithm.

Specifically, a deblurring rule based on a gradient distribution modelis determined based on a statistical properties analysis of an imagemodel and the gradient distribution of a blurry image and a definiteimage. The definite image meets a specific heavy-tailed distributionrule, and the blurry image does not meet the heavy-tailed distributionrule. A combined posterior probability of an original image and ablurring kernel is created during observing the original image. Theblurring kernel is obtained by maximizing the combined posteriorprobability, which indicates the motion direction of the pixel in theoriginal image.

In step S806, the motion direction of each of the pixels in the secondbase image other than the pixels with the same depth value iscalculated.

The motion direction of each of the pixels in the second base imageother than the pixels with the same depth value is calculated throughthe process similar to step S805.

As is shown in FIG. 9, which is a schematic diagram of the motiondirection of the depth image shown in FIG. 7. The curve shown in theblack box indicates the motion direction of the depth image.

In step S807, deblurring is performed on the first image based on thepreset deblurring rule in conjunction with the first motion direction toobtain the third image, and deblurring is performed on the second imagebased on the preset deblurring rule in conjunction with the secondmotion direction to obtain the fourth image.

Step S807 includes: performing deconvolution calculation on the pixel inthe first image to deblurr the first image to obtain the third imagewhich is already deblurred based on a preset blurring kernel model inconjunction with the first motion direction; and performing thedeconvolution calculation on the pixel in the second image to deblurrthe second image to obtain the fourth image which is already deblurredbased on the preset blurring kernel model in conjunction with the secondmotion direction.

The deconvolution calculation is performed on each of the pixels in thefirst image to deblurr the first image to obtain the definite thirdimage based on the preset deblurring rule in conjunction with theblurring kernel calculated in step S805 and the first motion direction.And the deconvolution calculation is performed on each of the pixels inthe second image to deblurr the second image to obtain the definitefourth image based on the preset deblurring rule in conjunction with theblurring kernel calculated in step S805 and the second motion direction.

For example, there is a known blurry image P(x, y), the definite imagecalculated based on the blurry image P(x, y) is represented as an imageI(x, y). The relationship between the two images is P(x, y)=I(x, y)*K.

In the above equation, * represents a convolution operation, K is theblurring kernel.

The equation is transformed as:I(x, y)=argmin∥P(x, y)−I(x, y)*K∥ ² +∥I(x, y)∥².

The above equation is solved by using the ROF (Rudin-Osher-Fatemi),blurring is performed on the blurry image to obtain the clear image.

The ROF (Rudin-Osher-Fatemi) is a known algorithm in the conventionalart, which is not described herein.

In step S808, the third image is combined with the fourth image based ona preset combining rule to obtain a fifth image.

Step S808 is the same as step S405 in the second embodiment of the imageprocessing method, which is not described herein.

In summary, according to the image processing method provided in theembodiment, the motion direction of the image is determined based on thepreset motion direction estimating algorithm in conjunction with thedepth image of the image, deblurring is performed on the image based onthe motion direction of the image, the images which are alreadydeblurred are combined to obtain a definite three-dimensionalperspective image. According to the image processing method, deblurringis performed on an image generated, the blurring caused by movement orthe like may be weakened, the definition of the deblurred image isbetter than the definition of the blurry image, thereby leading to abetter imaging result.

The image processing method is described in detail in the aboveembodiments of the disclosure, and the image processing method accordingto the disclosure may be implemented in various forms. It is alsoprovided an image processing apparatus according to the disclosure. Theembodiments of the image processing apparatus are described in detailbelow.

As is shown in FIG. 10, which is a schematic structural diagram of afirst embodiment of an image processing apparatus according to theapplication. The image processing apparatus is applied to an electronicdevice equipped with a binocular camera. The electronic device may be adesktop, a laptop, a tablet computer, a mobile phone, a smart TV, asmart watch, a wearable device or the like.

The image processing apparatus includes a receiving module 1001, acalculating module 1002, a deblurring module 1003 and a combining module1004.

The receiving module 1001 is configured to receive a first image and asecond image captured by the binocular camera.

One of two cameras of the binocular camera equipped in the electronicdevice captures a capture object for one frame, and the other one of thetwo cameras of the binocular camera captures simultaneously the samecapture object for one frame. The receiving module 1001 receives the twoframes of images and sets respectively the two frames of images as thefirst image and the second image.

The first image is corresponding to a position of the one of the twocameras of the binocular camera, and the second image is correspondingto a position of the other one of the two cameras of the binocularcamera. The first image and the second image are similar but different.

In practice, the images captured may be blurry due to that theelectronic device trembles or the capture object moves when capturingthe images.

FIG. 2A is the first image captured by the one of the two cameras of thebinocular camera, and FIG. 2B is the second image captured by the otherone of the two cameras of the binocular camera. The first image and thesecond image are blurry, content of the two images are similar butdifferent.

The calculating module 1002 is configured to calculate a motiondirection of the first image and a motion direction of the second imagebased on a preset motion algorithm.

First to be noted that an actual image blurring principle is thatdifferent depth values of the pixels correspond to different motiondirections of the pixels which cause the blurring. Therefore, the motiondirection of the blurry image is calculated, and deblurring is performedon the blurry image based on the motion direction.

The motion algorithm is preset, and the motion direction of the imagemay be calculated based on the preset motion algorithm.

Specifically, the calculating module 1002 calculates the motiondirection of the first image based on information of the first image andthe calculating module 1002 calculates the motion direction of thesecond image based on information of the second image.

The process of calculating the motion direction is described in detailin later embodiments, which is not described herein.

The deblurring module 1003 is configured to perform deblurring on thefirst image based on a preset deblurring rule in conjunction with themotion direction of the first image to obtain a third image, and performdeblurring on the second image based on the preset deblurring rule inconjunction with the motion direction of the second image to obtain afourth image.

The deblurring rule is preset, and the preset deblurring rule matcheswith the motion direction. The deblurring module 1003 may performdeblurring on the image the motion direction of which is alreadydetermined based on the preset deblurring rule to obtain a more definiteimage.

Specifically, blurring is performed on the first image based on thepreset deblurring rule in conjunction with the motion direction of thefirst image to obtain the third image, and blurring is performed on thesecond image based on the preset deblurring rule in conjunction with themotion direction of the second image to obtain the fourth image.

The process of performing deblurring on the image is described in detailin later embodiments, which is not described herein.

The combining module 1004 is configured to combine the third image withthe fourth image based on a preset combining rule to obtain a fifthimage.

The definition of the third image is better than the definition of thefirst image, and the definition of the fourth image is better than thedefinition of the second image.

The combining module 1004 is configured to: acquire a first pixel in thethird image and information about a position of the first pixel, anddetermine the capture object corresponding to the first pixel; searchfor a second pixel in the fourth image matching with the capture objectcorresponding to the first pixel based on the capture objectcorresponding to the first pixel, and determine information about aposition of the second pixel, where the first pixel and the second pixelconstitute a corresponding pixel group; calculate a position deviationof corresponding pixel group constituted of the first pixel and thesecond pixel based on a preset algorithm in conjunction with theinformation about the position of the first pixel and the informationabout the position of the second pixel; calculate the position deviationof each corresponding pixel group between the third image and the fourthimage; restore depth information in a three-dimensional coordinate basedon the position deviation of each corresponding pixel group; and combinethe corresponding pixels in the third image and the fourth image basedon the depth information of each corresponding pixel group to obtain thefifth image.

As is shown in FIG. 3, which shows the fifth image obtained by combiningthe first image and the second image in FIGS. 2A and 2B. The fifth imageis more definite than the first image and the second image, and may leadto a strong three-dimensional feeling.

In summary, it is provided an image processing apparatus according tothe embodiment. According to the image processing apparatus, a motiondirection of a first image and a motion direction of a second image arecalculated based on a preset motion algorithm, where the first image iscaptured by one of two cameras of the binocular camera and the secondimage is captured by the other one of the two cameras of the binocularcamera; deblurring is performed on the first image and the second imagebased on a preset deblurring rule; and the images which are alreadydeblurred are combined to obtain a clear three-dimensional perspectiveimage. According to the image processing apparatus, a blurred imagegenerated during imaging is deblurred, the blurring caused by movementor the like may be weakened, the sharpness of the deblurred image isbetter than the sharpness of the blurred image, thereby leading to abetter imaging result.

As is shown in FIG. 11, which is a schematic structural diagram of asecond embodiment of the image processing apparatus according to theapplication. The image processing apparatus includes a receiving module1101, a calculating module 1102, a deblurring module 1103 and acombining module 1104.

The calculating module 1102 includes a first analyzing unit 1105 and afirst calculating unit 1106.

The functions of the receiving module 1101, the deblurring module 1103and the combining module 1104 are respectively the same as the functionsof the receiving module 1001, the deblurring module 1003 and thecombining module 1004 in the first embodiment of the image processingapparatus, which are not described herein.

The first analyzing unit 1105 is configured to analyze the first imagebased on a preset depth analyzing rule to obtain a first depth image,and to analyze the second image based on the preset depth analyzing ruleto obtain a second depth image.

The depth refers to a spatial distance between the capture object in theimage and the camera by which the image is captured.

The depth analyzing rule is preset in the first analyzing unit 1105, thefirst image is analyzed based on the preset depth analyzing rule toobtain the first depth image of the first image, and the second image isanalyzed based on the preset depth analyzing rule to obtain the seconddepth image of the second image.

Specifically, analyzing the first image based on the preset depthanalyzing rule to obtain the first depth image includes: selecting apixel A from the pixels in the first image; searching for a pixel A′ inthe second image matching with the pixel A; determining an actualphysical position corresponding to the pixel A by using thetriangulation algorithm based on a position of the pixel A in the firstimage, a position of the pixel A′ in the second image and a spatialparameter of the binocular camera; calculating a depth value of thepixel A in the first image based on the distance between the actualphysical position corresponding to the pixel A and the position of thepixel A in the first image; calculating the depth value of each of thepixels in the first image other than the pixel A; and drawing the firstdepth image based on the depth value of each of the pixels in the firstimage.

Specifically, analyzing the second image based on the preset depthanalyzing rule to obtain the second depth image includes: selecting apixel B from the pixels in the second image; searching for a pixel B′ inthe first image matching with the pixel B; determining an actualphysical position corresponding to the pixel B by using thetriangulation algorithm based on a position of the pixel B′ in the firstimage, a position of the pixel B in the second image and the spatialparameter of the binocular camera; calculating a depth value of thepixel B in the second image based on the distance between the actualphysical position corresponding to the pixel B and the position of thepixel B in the second image; calculating the depth value of each of thepixels in the second image other than the pixel B; and drawing thesecond depth image based on the depth value of each of the pixels in thesecond image.

The spatial parameters of the binocular camera include the distancebetween the centers of two cameras or the like.

It should be noted that the method for calculating the depth image ofeach image in the embodiment is not limited to the described method. Inpractice, the method for calculating the depth image may be othermethods capable of obtaining the depth image of the image, which is notlimited herein.

The first calculating unit 1106 is configured to estimate the firstdepth image based on a preset estimating algorithm to obtain a firstmotion direction of a pixel in the first image, and to estimate thesecond depth image based on the preset estimating algorithm to obtain asecond motion direction of a pixel in the second image.

Each pixel in the image corresponds to a depth value, the depth value isa comparable value, the depth values of the pixels are compared todetermine whether the pixels are the same pixel.

The estimating algorithm is preset in the first calculating unit 1106,and generally the preset estimating algorithm is the blind deconvolutionalgorithm.

Specifically, the motion direction may be calculated based on the pixelswith the same depth value, the process of calculating the motiondirection is described in detail in later embodiments, which is notdescribed herein.

In summary, according to the image processing apparatus provided in theembodiment, the depth image of the first image and the depth image ofthe second image are determined based on the preset depth analyzingrule, the first motion direction of the pixel in the first image and thesecond motion direction of the pixel in the second image are obtainedbased on the preset estimating algorithm in conjunction with the depthimages, and deblurring is performed on the blurry image based on themotion direction of blurry image. According to the image processingapparatus, the motion direction of the pixels in the image is calculatedbased on the depth image of the image, deblurring is performed on ablurry image generated during imaging, the blurring caused by movementor the like may be weakened, the definition of the deblurred image isbetter than the definition of the blurry image, thereby leading to abetter imaging result.

It is provided a third embodiment of the image processing apparatusaccording to the application. The image processing apparatus includes areceiving module, a calculating module, a deblurring module and acombining module.

The calculating module includes a first analyzing unit and a firstcalculating unit.

As is shown in FIG. 12, which is a schematic structural diagram of thefirst analyzing unit in the third embodiment of the image processingapparatus according to the application. The first analyzing unitincludes a first selecting subunit 1201, a searching subunit 1202, afirst calculating subunit 1203 and a drawing subunit 1204.

The functions of the receiving module, the deblurring module, thecombining module and the first calculating unit in the third embodimentof the image processing apparatus are respectively the same as thefunctions of the receiving module 1101, the deblurring module 1103, thecombining module 1104 and the first calculating unit 1106 in the secondembodiment of the image processing apparatus, which are not describedherein.

The first selecting subunit 1201 is configured to select any one of thefirst image and the second image as a first base image with the otherone as a first reference image, and to select a pixel from the pixels inthe first base image.

The first selecting subunit 1201 selects any one of the first image andthe second image as the first base image, and the other one is as thefirst reference image. A motion direction of the pixel in the first baseimage is calculated based on the first base image and the firstreference image.

Each image includes a plurality of pixels, and any one of the pixels inthe first base image is selected as a base pixel for determining themotion direction.

It should be noted that the searching subunit 1202 and the firstcalculating subunit 1203 are used to calculate the motion direction ofany one of the pixels in the first base image.

The searching subunit 1202 is configured to search for a correspondingpixel in the first reference image matching with the selected pixel.

The first base image and the first reference image are captured by twocameras of the binocular camera, the two images are similar but there isa little angle difference between the two images. In each of the twoimages, there is image content corresponding to the same capture object.

Therefore, the searching subunit 1202 searches for a pixel in the firstreference image corresponding to the selected pixel, a position of thecorresponding pixel in the first reference image is different from aposition of the selected pixel in the first base image.

The process performed by the searching subunit 1202 includes:determining a capture object corresponding to the selected pixel in thefirst base image; the corresponding pixel in the first reference imagematching with the selected pixel is searched based on the captureobject, and the matching pixel is set as the corresponding pixelmatching with the selected pixel.

Specifically, in the case that the selected pixel and the correspondingpixel matching with the selected pixel are for the same capture object(or capture content), a matching degree between the selected pixel andthe corresponding pixel is high.

In practice, the process of determining the matching degree between theselected pixel and the corresponding pixel may include: firstlydetermining the position of the selected pixel in the first base image,and searching for a region in the first reference image close to theposition of the selected pixel in the first base image based on theposition of the selected pixel in the first base image; and thencomparing each of the pixels in the region with the selected pixel inthe first base image based on various information about the capturecontent such as color of the pixel or the like, to obtain the matchingdegree, the pixel with the highest matching degree is the correspondingpixel matching with the selected pixel.

The first calculating subunit 1203 is configured to determine a depthvalue of the selected pixel by using a preset depth algorithm based on aposition of the selected pixel in the first base image, a position ofthe corresponding pixel in the first reference image and a spatialparameter of the binocular camera, and to calculate the depth value ofeach of the pixels in the first base image other than the selectedpixel.

The position of the selected pixel in the first base image such as acoordinate position of the selected pixel in the first base image isrecorded when the selected pixel is selected.

The first calculating subunit 1203 determines the position, such as acoordinate position, of the corresponding pixel in the first referenceimage determined by the searching subunit 1202.

The depth algorithm is preset, and an actual physical positioncorresponding to the selected pixel is calculated by using the presetdepth algorithm based on the position of the selected pixel in the firstbase image, the position of the corresponding pixel in the firstreference image and the spatial parameter of the binocular camera. Theactual physical position corresponding to the selected pixel refers tothe position of the capture object corresponding to the selected pixelin three-dimensional space.

As is shown in FIG. 6, which is a schematic diagram of calculating theactual physical position corresponding to the pixel. C1 and C2 arepositions of two cameras of the binocular camera, in an image 601, apixel x1 is a certain pixel in the image captured by the camera at C1,and in an image 602, a pixel x2 is a certain pixel in the image capturedby the camera at C2, the pixel x1 and the pixel x2 are pixels for acapture object X, the capture object X locates at an intersection pointbetween a straight line defined by C1 and the pixel x1 and a straightline defined by C2 and the pixel x2, the distance between C1 and C2 isknown, the distance between C1 and the pixel x1 is determined when thepixel x1 is selected, and the distance between C2 and the pixel x2 isdetermined when the pixel x2 is selected. The distance between thecapture object X and the pixel x1 and the distance between the captureobject X and the pixel x2 may be calculated according to thetrigonometric function. The actual physical position corresponding tothe pixel x1 is calculated based on the coordinates of C1, C2, the pixelx1 and the pixel x2.

Specifically, the actual physical position refers to the distancebetween the capture object X and the electronic device, such as thedistance between the capture object X and the midpoint of C1 and C2.

It should be noted that in the case that an xy coordinate system isdefined with the straight line defined by C1 and C2 being an x-axis andthe direction from C1 to C2 being a positive direction of the x-axis andthe direction from a point in the straight line defined by C1 and C2 tothe capture object X being a positive direction of a y-axis, thecoordinate of the capture object X may be determined as the actualphysical position corresponding to the pixel x1.

It should be noted that the actual physical position corresponding tothe pixel x1 is the same as the actual physical position correspondingto the pixel x2 in the image 602. In the case that the image 602 is setas the first base image, the actual physical position corresponding tothe pixel x2 is not required to be calculated, because the actualphysical position corresponding to the pixel x2 is the same as theactual physical position corresponding to the pixel x1 calculated basedon the image 601.

The depth value of the pixel x1 refers to the distance between thecapture object X corresponding to the pixel x1 and the camera forcapturing the image to which the pixel x1 belongs.

The actual physical position corresponding to the pixel x1 isdetermined, the distance between the actual physical positioncorresponding to the pixel x1 and the camera for capturing the image towhich the pixel x1 belongs is calculated based on the actual physicalposition corresponding to the pixel x1 and the position of the camerafor capturing the image to which the pixel x1 belongs, thereby the depthvalue of the pixel x1 is calculated.

As is shown in FIG. 6, in the case that the distance between the captureobject X to the straight line defined by C1 and C2 is known, thedistance between the capture object X to C1 may be calculated accordingto the triangulation algorithm, thereby the depth value of the pixel x1in the first base image is calculated.

The first calculating subunit 1203 calculates the depth value of any oneof the pixels in the first base image, and similarly calculates thedepth value of each of the pixels in the first base image other than theselected pixel.

It should be noted that, the first image is selected as the first baseimage and the depth value of each of the pixels in the first base imageis calculated. In the case that the second image is selected as thefirst base image, if there is a pixel in the second image matching witha pixel in the first image, the actual physical position correspondingto the pixel in the second image is not required to be calculated,because the actual physical position corresponding to the pixel in thesecond image is the same as the actual physical position correspondingto the matching pixel in the first image.

The drawing subunit 1204 is configured to draw the depth image of thefirst base image based on the depth value of each of the pixels in thefirst base image.

The depth value of each of the pixels in the first base image isdetermined by the first calculating subunit 1203. The drawing subunit1204 draws the depth image of the first base image based on the depthvalue of each of the pixels in the first base image.

Specifically, a gray value of the pixel is used to indicate the depthvalue of the pixel, the greater the depth value of the pixel is, thegreater the gray value of the pixel is.

FIG. 7 shows a first depth image of the first image in FIG. 2A.

The first image is taken as an example, the process performed by thefirst selecting subunit 1201, the searching subunit 1202, the firstcalculating subunit 1203 and the drawing subunit 1204 includes:selecting a pixel A from the pixels in the first image; searching for apixel A′ in the second image matching with the pixel A; determining anactual physical position corresponding to the pixel A by using thetriangulation algorithm based on a position of the pixel A in the firstimage, a position of the pixel A′ in the second image and a spatialparameter of the binocular camera; calculating a depth value of thepixel A in the first image based on the distance between the actualphysical position corresponding to the pixel A and the position of thepixel A in the first image; calculating the depth value of each of thepixels in the first image other than the pixel A; and drawing the firstdepth image based on the depth value of each of the pixels in the firstimage.

The second image is taken as an example, the process performed by thefirst selecting subunit 1201, the searching subunit 1202, the firstcalculating subunit 1203 and the drawing subunit 1204 includes:selecting a pixel B from the pixels in the second image; searching for apixel B′ in the first image matching with the pixel B; determining anactual physical position corresponding to the pixel B by using thetriangulation algorithm based on a position of the pixel B′ in the firstimage, a position of the pixel B in the second image and the spatialparameter of the binocular camera; calculating a depth value of thepixel B in the second image based on the distance between the actualphysical position corresponding to the pixel B and the position of thepixel B in the second image; calculating the depth value of each of thepixels in the second image other than the pixel B; and drawing thesecond depth image based on the depth value of each of the pixels in thesecond image.

In summary, according to the image processing apparatus provided in theembodiment, the depth value of each of the pixels in the image iscalculated, the depth image of the image is drawn, the motion directionof the pixel in the image is determined based on the depth image,deblurring is performed on the image, the images which are alreadydeblurred are combined to obtain a clear three-dimensional perspectiveimage. According to the image processing apparatus, deblurring isperformed on a blurry image generated during imaging, the blurringcaused by movement or the like may be weakened, the definition of thedeblurred image is better than the definition of the blurry image,thereby leading to a better imaging result.

It is provided a fourth embodiment of the image processing apparatusaccording to the application. The image processing apparatus includes areceiving module, a calculating module, a deblurring module and acombining module.

The calculating module includes a first analyzing unit and a firstcalculating unit.

As is shown in FIG. 13, which is a schematic structural diagram of thefirst calculating unit in the fourth embodiment of the image processingapparatus according to the application. The first calculating unitincludes a second selecting subunit 1301, a determining subunit 1302 anda second calculating subunit 1303.

The functions of the receiving module, the deblurring module, thecombining module and the first analyzing unit in the fourth embodimentof the image processing apparatus are respectively the same as thefunctions of the receiving module 1101, the deblurring module 1103, thecombining module 1104 and the first analyzing unit 1105 in the secondembodiment of the image processing apparatus, which are not describedherein.

The second selecting subunit 1301 is configured to select any one of thefirst image and the second image as a second base image, and to set thedepth image of the second base image as a base depth image.

The second selecting subunit 1301 selects any one of the first image andthe second image as the second base image and sets the depth image ofthe second base image as the base depth image. The motion direction ofthe pixel in the second base image is calculated based on the secondbase image and the base depth image of the second base image.

It should be noted that a process performed by the determining subunit1302 and the second calculating subunit 1303 is the process ofcalculating the motion direction of any one of the pixels in the secondbase image.

The determining subunit is configured to determine pixels with the samedepth value in the second base image and positions of the pixels withthe same depth value in the second base image based on the base depthimage.

An actual image blurring principle is that different depth values of thepixels correspond to different motion directions of the pixels whichcause the blurring. In the present embodiment, the motion direction ofthe pixel is calculated based on the depth value of the pixel, andspecifically, the motion direction of the pixel is calculated based onthe pixels with the same depth value.

Specifically, the determining subunit 1302 determines the pixels withthe same depth value in the second base image and positions of thepixels with the same depth value in the second base image based on thebase depth image. The pixels with the same depth value in the secondbase image constitute a pixel group with the same depth value, and thepixel group includes a plurality of pixels.

The second calculating subunit is configured to calculate the motiondirection of the pixels with the same depth value in the second baseimage based on a preset motion direction estimating algorithm inconjunction with information about the positions of the pixels with thesame depth value in the second base image, and to calculate the motiondirection of each the pixels in the second base image other than thepixels with the same depth value.

The preset motion direction estimating algorithm is the blinddeconvolution algorithm.

Specifically, a deblurring rule based on a gradient distribution modelis determined based on a statistical properties analysis of an imagemodel and the gradient distribution of a blurry image and a definiteimage. The definite image meets a specific heavy-tailed distributionrule, and the blurry image does not meet the heavy-tailed distributionrule. A combined posterior probability of an original image and ablurring kernel is created during observing the original image. Theblurring kernel is obtained by maximizing the combined posteriorprobability, which indicates the motion direction of the pixel in theoriginal image.

The motion direction of each of the pixels in the second base imageother than the pixels with the same depth value is calculated similarly.

As is shown in FIG. 9, which is a schematic diagram of the motiondirection of the depth image shown in FIG. 7. The curve shown in theblack box indicates the motion direction of the depth image.

The deblurring module is configured to: perform deconvolutioncalculation on the pixel in the first image to deblurr the first imageto obtain the third image which is already deblurred based on a presetblurring kernel model in conjunction with the first motion direction;and to perform the deconvolution calculation on the pixel in the secondimage to deblurr the second image to obtain the fourth image which isalready deblurred based on the preset blurring kernel model inconjunction with the second motion direction.

The deblurring module performs the deconvolution calculation on each ofthe pixels in the first image to deblurr the first image to obtain thedefinite third image based on the preset deblurring rule in conjunctionwith the blurring kernel calculated by the second calculating subunit1303 and the first motion direction. And the deblurring module performsthe deconvolution calculation on each of the pixels in the second imageto deblurr the second image to obtain the definite fourth image based onthe preset deblurring rule in conjunction with the blurring kernelcalculated by the second calculating subunit 1303 and the second motiondirection.

For example, there is a known blurred image P(x, y), the clear imagecalculated based on the blurred image P(x, y) is represented as an imageI(x, y). The relationship between the two images is P(x, y)=I(x, y)*K.

In the above equation, * represents a convolution operation, K is theblurring kernel.

The equation is transformed as:I(x, y)=argmin∥P(x, y)−I(x, y)*K∥ ² +∥I(x, y)∥².

The above equation is solved by using the ROF (Rudin-Osher-Fatemi),deblurring is performed on the blurry image to obtain the definiteimage.

The ROF (Rudin-Osher-Fatemi) is a known algorithm in the conventionalart, which is not described herein.

In summary, according to the image processing apparatus provided in theembodiment, the motion direction of the image is determined based on thepreset motion direction estimating algorithm in conjunction with thedepth image of the image, deblurring is performed in the image based onthe motion direction of the image, the images which are alreadydeblurred are combined to obtain a definite three-dimensionalperspective image. According to the image processing apparatus,deblurring is performed on an image generated, the blurring caused bymovement or the like may be weakened, the definition of the deblurredimage is better than the definition of the blurry image, thereby leadingto a better imaging result.

It is provided an image processing apparatus according to theapplication, and correspondingly, it is also provided an electronicdevice according to the application. The electronic device includes abinocular camera and any one of the image processing apparatusesaccording to the above embodiments.

The image processing apparatuses includes a receiving module, acalculating module, a deblurring module and a combining module. Thefunctions of various modules of the image processing apparatuses arerespectively the same as the functions of the corresponding modules ofany one of the image processing apparatuses according to the aboveembodiments, which are not described herein.

Optionally, the calculating module includes a first analyzing unit and afirst calculating unit. The functions of various modules of the imageprocessing apparatuses are respectively the same as the functions of thecorresponding modules of any one of the image processing apparatusesaccording to the above embodiments, which are not described herein.

Optionally, the first analyzing unit includes a first selecting subunit,a searching subunit, a first calculating subunit and a drawing subunit.The functions of various modules of the image processing apparatuses arerespectively the same as the functions of the corresponding modules ofany one of the image processing apparatuses according to the aboveembodiments, which are not described herein.

Optionally, the first calculating unit includes a second selectingsubunit, a determining subunit and a second calculating subunit. Thefunctions of various modules of the image processing apparatuses arerespectively the same as the functions of the corresponding modules ofany one of the image processing apparatuses according to the aboveembodiments, which are not described herein.

Optionally, the deblurring module is configured to perform deconvolutioncalculation on the pixel in the first image to deblur the first image toobtain the third image which is already deblurred based on a presetblurring kernel model in conjunction with the first motion direction;and to perform the deconvolution calculation on the pixel in the secondimage to deblur the second image to obtain the fourth image which isalready deblurred based on the preset blurring kernel model inconjunction with the second motion direction.

The embodiments of the present disclosure are described herein in aprogressive manner, with an emphasis placed on explaining the differencebetween each embodiment and the other embodiments; hence, for the sameor similar parts among the embodiments, they can be referred to from oneanother. For the apparatus provided in the embodiments, thecorresponding description is relatively simple because the apparatuscorrespond to the method provided in the embodiments. The relevantportions may be referred to the description for the method parts.

The above description of the embodiments provided herein enables thoseskilled in the art to implement or use the present disclosure. Numerousmodifications to the embodiments will be apparent to those skilled inthe art, and the general principle herein can be implemented in otherembodiments without deviation from the spirit or scope of the presentdisclosure. Therefore, the present disclosure will not be limited to theembodiments described herein, but in accordance with the widest scopeconsistent with the principle and novel features provided herein.

The invention claimed is:
 1. An image processing method, comprising:capturing a first image and a second image using an imaging device;analyzing the first image based on a preset depth analyzing rule toobtain a first depth image, and analyzing the second image based on thepreset depth analyzing rule to obtain a second depth image: estimatingthe first depth image based on a preset estimating algorithm to obtain afirst motion direction of a pixel in the first image, and estimating thesecond depth image based on the preset estimating algorithm to obtain asecond motion direction of a pixel in the second image; performingdeblurring on the first image based on the motion direction of the firstimage to obtain a third image, and performing deblurring on the secondimage based on the motion direction of the second image to obtain afourth image; and combining the third image with the fourth image toobtain a fifth image.
 2. The image processing method according to claim1, wherein analyzing the first image based on the preset depth analyzingrule to obtain the first depth image and analyzing the second imagebased on the preset depth analyzing rule to obtain the second depthimage comprises: selecting one of the first image and the second imageas a first base image with the other one as a first reference image;selecting a pixel from the pixels in the first base image; searching fora corresponding pixel in the first reference image that matches with theselected pixel in the first base image; determining a depth value of theselected pixel by using a preset depth algorithm based on a position ofthe selected pixel in the first base image, a position of thecorresponding pixel in the first reference image and a spatial parameterof the imaging device; calculating the depth value of each of the pixelsin the first base image other than the selected pixel; and deriving adepth image of the first base image based on the depth value of each ofthe pixels in the first base image.
 3. The image processing methodaccording to claim 1, wherein estimating the first depth image based onthe preset estimating algorithm to obtain the first motion direction ofa pixel in the first image and estimating the second depth image basedon the preset estimating algorithm to obtain the second motion directionof a pixel in the second image comprises: selecting one of the firstimage and the second image as a second base image and setting the depthimage of the second base image as a base depth image; determining pixelswith the same depth value in the second base image and positions of thepixels with the same depth value in the second base image based on thebase depth image; calculating the motion direction of the pixels withthe same depth value in the second base image based on a preset motiondirection estimating algorithm in conjunction with information about thepositions of the pixels with the same depth value in the second baseimage; and calculating the motion direction of each of the pixels in thesecond base image other than the pixels with the same depth value. 4.The image processing method according to claim 3, wherein performingdeblurring on the first image based on the preset deblurring rule inconjunction with the first motion direction to obtain the third imageand performing deblurring on the second image based on the presetdeblurring rule in conjunction with the second motion direction toobtain the fourth image comprises: performing deconvolution calculationon the pixel in the first image to obtain the third image based on apreset blurring kernel model in conjunction with the first motiondirection; and performing the deconvolution calculation on the pixel inthe second image to obtain the fourth image based on the preset blurringkernel model in conjunction with the second motion direction.
 5. Animage processing apparatus, comprising: an imaging device for capturinga first image and a second image, a processor, and a memory storingprocessor-executable instructions, wherein the instructions, whenexecuted by the processor, configure the processor to: analyze the firstimage based on a preset depth analyzing rule to obtain a first depthimage, and to analyze the second image based on the preset depthanalyzing rule to obtain a second depth image; estimate the first depthimage based on a preset estimating algorithm to obtain a first motiondirection of a pixel in the first image, and to estimate the seconddepth image based on the preset estimating algorithm to obtain a secondmotion direction of a pixel in the second image; perform deblurring onthe first image based on the motion direction of the first image toobtain a third image, and perform deblurring on the second image basedon the motion direction of the second image to obtain a fourth image;and combine the third image with the fourth image to obtain a fifthimage.
 6. The image processing apparatus according to claim 5, whereinthe processor is configured to: select one of the first image and thesecond image as a first base image with the other one as a firstreference image, and to select a pixel from the pixels in the first baseimage; search for a corresponding pixel in the first reference imagematching with the selected pixel; determine a depth value of theselected pixel by using a preset depth algorithm based on a position ofthe selected pixel in the first base image, a position of thecorresponding pixel in the first reference image and a spatial parameterof a binocular camera, and calculate the depth value of each of thepixels in the first base image other than the selected pixel; and drawthe depth image of the first base image based on the depth value of eachof the pixels in the first base image.
 7. The image processing apparatusaccording to claim 5, wherein the processor is configured to: select oneof the first image and the second image as a second base image, and toset the depth image of the second base image as a base depth image;determine pixels with the same depth value in the second base image andpositions of the pixels with the same depth value in the second baseimage based on the base depth image; and calculate the motion directionof the pixels with the same depth value in the second base image basedon a preset motion direction estimating algorithm in conjunction withinformation about the positions of the pixels with the same depth valuein the second base image, and calculate the motion direction of each thepixels in the second base image other than the pixels with the samedepth value.
 8. The image processing apparatus according to claim 7,wherein the processor is configured to: perform deconvolutioncalculation on the pixel in the first image to obtain the third imagebased on a preset blurring kernel model in conjunction with the firstmotion direction; and perform the deconvolution calculation on the pixelin the second to obtain the fourth image based on the preset blurringkernel model in conjunction with the second motion direction.
 9. Theimage processing apparatus according to claim 5, wherein the imagingdevice is a binocular camera.
 10. The image processing apparatusaccording to claim 9, wherein the binocular camera comprises a first anda second cameras, the first camera being configured to capture the firstimage and the second camera being configured to capture the secondimage.
 11. An electronic device comprising: a binocular camera forcapturing images, a processor, and a memory storing processor-executableinstructions, wherein the instructions, when executed by the processor,configure the processor to: receive a first image and a second imagecaptured by the binocular camera; analyze the first image based on apreset depth analyzing rule to obtain a first depth image, and toanalyze the second image based on the preset depth analyzing rule toobtain a second depth image; estimate the first depth image based on apreset estimating algorithm to obtain a first motion direction of apixel in the first image, and to estimate the second depth image basedon the preset estimating algorithm to obtain a second motion directionof a pixel in the second image; perform deblurring on the first imagebased on the motion direction of the first image to obtain a thirdimage, and perform deblurring on the second image based on the motiondirection of the second image to obtain a fourth image; and a combiningmodule configured to combine the third image with the fourth image toobtain a fifth image.